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ABSTRACT . ' 

The Education Amendments of 1976, Public Law 94-482, 
which revised the Vocational Education Act (VEA) of 1963, inrtroduced 
major new requirements for the evaluation of federally funded " 
vocational education programs. This paper describes the -activities 
stimulated by these evaluation requirements and shows how evaluation 
results. are used to revise and improve programs. However, since 
policy intentions-"in this case, program revision and ^ 
iinprovefflent--are not realized instantaneously, the paper examines^^^ 
number of intervening steps in the implementation process before 
considering outcomes. Before the outcomes can be realized, for 
example, the message from Congress must be transmitted and 
implemented at federal, state, and local levels. Another purpose o'f 
this paper is to describe the way in which this message was 
trarsm^itted through the Office of Education to the states 
to the -local level. (KG) 
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Introduction 

The Education Amendments of 1&76, P.L, 94-482, 
which revised the Vocational Education Act (VEA) ci 1963, 
introduced major new requi rern^eTits for the evaluat ion of 
federally funixed vocational tsdueat i on programs • Tjne of 
the reasons :cr the changes was a 1974 report, z the 
General Accounting Office (GAQ) on the implementation of 
Federal vocational education policy in seven St^ies ^ 
The results of this study, which was critical c: the 
States, were presented to Congress during reauthorization 
hearings for the VEA and were substantiated by other 
studies and testimony. The exisiting legislation i - 
eluded the general requi^rement that Federal funds were lo 
be distributed with "due consideration • • • to the 
results of periodic evaluations of St^ate and local voca- 
tional education programs, services and activities : 
light '^of information regarding current and projects 
manpower needs and job opportunities. . . (P.L. 90- 

576, Section 123(a)(6)(A)). However, the testimony pre- 
sented during the hearings indicated that funds wer 
distributed with little consideration of evaluatio 
results, in part because of "the lack of systemati 
program eva luat i on . "^^ ; 

One of the purposes of the Federal vocat ional eduea 
tion program is to assist States in offering programs c 
vocational education which are, among other things 
"realistic in light of actual or anticipated opportu- 
nities for gainful employment. . . ." (from the Declare 
tion of Purpose, Sec. - 101). Despite this purpose 
Congress was told that much of the pr esent enr o 1 Imen t in 
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One purpose of this paper is to describe the activ- 
ities stimulated by these evaluation r ecu : rerrjepts and to 
show how evaluat ion resul ts are used to -evise and im- 
:)rove programs. However, since policy : n ten t i ons-- i n 
:nis case, program revision and impro v^-^r^en t~-ar e not 
ralized instantaneously, it is necessary to examine a 
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The Evaluation Requirements 

The new eva lna t i on requi rements of the 1976 
Education Amendments (Sec. 112) were intended to promote 
rational planning and the responsiveness of the voca- 
tional education system to changing labor markets. The 
most substantial of these requirements originated in the 
House Bill, which stipulated that States evaluate all 
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federally ass i s ted - programs on a resriilar basis anc ex- 
amine job placement rates and employers' :opini=pjis, of 
^graduates of vocational education pr-^rgrams . They pro- 
"vide that: 
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(A) each State shall, dnrriirg the five-year 
period of the State plan, evaluals;- the ef feet ivemeijs 
of each program within the State :z ing assisted with 
iunds available -under this Aet; in d the results of 
these evaluations shall be use to revise the 
State's pr,ograms, and shall be 732 je readily avail- 
able to the State advisory counc. and 

(B) each State shall evaloiE.e, by using data 
collected, wherever possible, by s atist icai ly val id 
sampling techniques, each such p-ogivam within the^x 
State which purports to impart entry level job 
skills according 'to the exter- to which program" 
completers and leavers-- 

( r) find employment i:: occupations leiated 
to their training, and 

(ii) are considered by their employers to 
be well-trained and prepared for employment. . 
. . (Sec. 112(b)(1)). 

The emphasis given in (B) to the employmenl: success 
of program completers and leavers underscores, the contin- 
uing congressional concern, dating back to the original 
Smith-Hughes Act of 1.917, that federally supported voca- 
tional education should prepare students for participa- 
tion i n the labor force. According to the House Report, 
"The Committee has specified these two criteria to judge 
the effectiveness of these programs because in our 
opinion they show most clearly whether persons trained in 
vocational programs are showing the results of such 
training."^ These criteria are appropriate only- for- 
students in "occupational" programs which prepare them 
for entry-level jobs. "Nonoccupational" programs, such 
as^jndustr ial ..art!:s, consumer and homemaking education. 



i 

/ 

and prevail, t ona 1' pr-o^fframs , are. to be evaluat-ed as well, 
according: i: (A) above, but criteria for evaluation of 
these prcg::r:_::.3 are unspecified. 

Purposes dhJ Hvaluation 

• . * 

The Legislation envisions two purposes for evalua- 
tion: flrit, and most important, management and im- 
provement of programs; and, second, ^ national reporting. 
Further., the legislation, together with the House and 
Senate Eeports, gives a good indication of the sort of 
evaluation findings that were needed and the decisions 
that would flow from them.^ Congress intended that deci- 
sions on which programs to support with Federal funds be 
influenced by evaluation results, spec .* f i ea 1 ly results 
showing whether or not students from given program were 
finding jobs in related fields and whether or not their 
employers were satisfied with th'^em.-''^ The Congress 
seemed to want to produce a scenario like the 
following: evaluation results would indicate which 
programs were successful in placing students; these find- 
ings, in turn, would indicate which skills were in demand, 
which would then indicate which programs to continue. 

. The legislation does not -imply that placement data are to 
be the sole determinants o.f the decision to continue or 
discontinue a program or that this i's the only use to 
which evaluation data can be put. It does, however, seem 

. to suggest that low placement rates for particular 
programs would act as red flags s i gna 1 1 i ng t he need for a 
cilqser look. 

The leg i s lat i on encourages enactment of this sce- 
nario in several places. For example, it requires that 

-5- . 
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Federal funds be distributed, to eligibJe recipients on 
the basis, of local app 1 i ca t i orvs that descr ibe , among, 
other things, "how the findings of any evaluations o7 
• • programs. . .have been used to dev-etbp.,;t he program pro- 
posed in the applr-cation" (Section 108 ( a ) ( 4 ) (3) ( i i) ) . 
In addition, the State's annual accountability report 
. must "contain a summary of the evaluations of programs 
required to be conducted by section 112 and a description 
of how the information from .these evaluations has been, 
or.. -Is being, used by, the State board to improve its 
. programs" ^Section 108(b)(2)(C)). ' ;~ 

National reporting", a second, use /of evaluation 
results called for by the, legislation," deserves 
cormient. Student placement rates are to be entered into 
the Vocational .. Education Data System (VEDS) and aggre- 
gated across localities and States to . form a national 
' picture (Section 163(a)(1)). The two uses may not always 
be compatible; information needed for one set of purposes 
may be d i f f e r en t f r om that needed for another, as may the ' 
requirements for s t a t i s t i ca 1 r i gor . F6r example, program 
improvement decisions at the local level can be based on 
^ fairly "dirty"..,.or -soft" data, collected in alorm not 
comparable across localities. On the. other hand, consid- 
erable standardization is necessary for placement rate 
data, which are collected at the school level and aggre- 
gated successive'ly at the district, the, State," and 

levels to provide a' natT^^^^^^^^ on the ef- . 

fectiveness of vocational education. 
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Reaction from the Field 



State' and local vocational education directors ob- 
jected strenuously to the evaluation requirements. Their 
objections centered on using job placement, which they 
viewed as unduly narrow, as the' major criterion of ef- 
fectiveness. First, they argued, it does not adequately 
reflect the nature' or goals of vocational education,, 
which is a broad educat i onal prografn, not just' a training 
program. Second , vocat ional educators ins isted that- the 
criterion of job placement holds schools to an inappro- 
priate standard. School systems should be held account- 
able for imparting certain knowledge and skills and for 
making certain that there is a good fit between what is 
taught and the jobs available. However, what jobs 
students' actually take is determined by a hosi of 
economic and personal factors beyond the school's 
control..^ "Hold us a'bcc^untabie for employabi 1 itj rc but not 
employment , " is the way many vocat i ona 1 educators phrased 
J:he concern. 

Beyond these two objections, vocational educators 
voiced a fear about the possible consequences of evalu- 
ating programs only, in terms of employment and employers' 
satisfaction, especially if funding decisions flowed from 
the results. Easy- t'o-place students, they suggested, 
might be accepted for programs at the expense of those 
har de r . to' place, such as mertib.e r s of minority groups , the 
disadvantaged, or women in nont radi t iona 1 programs-- in 
short, the yery groups Congress most wanted to reach. 



The Office of Education Interprets the Requirements 



The Office of Education (OE) played an important 
role in seeing that States understood and were able to 
car^'y out Congress' message. It /^oth explicated the 
provisions of the Act anT oTrereTi 

them out. Through . regulat ions , p^b4^ and the 

like, OE interpreted the pr ov i s i on^^^oi^ the Act. An exam- 
ination of the contents of these documents revealed the 
way in which the Congressional message was transmitted to 
the States . 

\ , 

The Offi^Nof Education interpreted the evaluation 
requirements through regulations issued October 3, 197',7-- 
one year after the Act was signed--and a policy memor- 
andum issued April 2.4, i979--one and one-half years 
later. Thus, it took the OE two and one-half years to 
get a full set of instructions to the States. In the 
meantime, the States were already implementing the legis- 
lation and, in-some cases, had to change th-ei r ' procedures 
after receiving official guidance. Thus, the timing of 
guidance alone affected the orderfiness and faithfulness 
of implementation of legis lat ion. ^ 

The content of the guidance affected faithfulness of^ 
implementation in a/ more obvious way. With respect to 
cojUent^ the regulations, the first guidance from OE, 
extended ~Fhe reqinTemen"t~s the Act substantially. 

While the statute referred only to federally supported 
programs, the regulations require that programs to be 
evaluated include those supported by Federal, State, and 
local funds. Furthermore, the regulations broaden con- 
siderably the dimensions along which the programs are to 



be evaluated, in part in response to the objections from 
the field and from a belief that placement rates by them- 
selves could give a false impression of program 
problems. It is important to note that the regulations 
give guidance on procedure only; the requirement in the 
1 e g i s 1 a t i on _ t h a t _e v a 1 u_a J J q n^^^ be used to revise and 
improve programs is not developed. The interpretation 
does not suggest, for example, what type of information 
might be used or how it might be used to improve 
programs. The areas to be evaluated are: / 

a. planning and operational processes; 

b. student achievement; 

c. student employment success; 

d. the effects of additional services, as measured 
by the suggested criteria under (a),(b), and 
(c), that the State provides to special popula- 
tions. 

Within each area, the regulations also suggest criteria 
to be used (e.g., wage rates and duration of employment) 
as indicators of student employment success, as shown in 
Table 1. 

Since the regulations include, student employment 
success as just one of four sets of criteria against 
which programs should be measured, they might seem more 
acceptable to vocational educators than would a provision 
requiring evaluation on the basis of student employment 
alone. As a pract ical mat te r , however, the .. regulat i^ons 
have been a source of dissatisfaction. State administra- 
tors argue that they do not have the resources to imple- 
ment all of the evaluation, requirements, ^nd they are not 



SPECIFICATIONS IN THE REGULATIONS OF 
EVALUATION BY STATE BOARDS ^ 



The State board shall, during the five-year per 
of the State' plan, evaluate in quantitative terms 
effectiveness of each formally organized program 
project supported by Federal, State, and local fun 
These eva luat i ons shall be in terms of: 

(a) Planning and operational processes, such as: 

"~(l)~QuaHty and availability of instructional 

offerings J 

(2) Guidance, counseling, and placement and 
follow-up services; 

(3) Capacity and condition of facilities and 
equipment; 

(4) Biployer participation in cooperative prograns of 
vocational education; 

(5) Teacher/pupil ratios; and 

(6) Teacher qualifications. 

(b) Results of student achievement as measured, for 
exannple, by: 

(1) Standard occupational prof iciency measures; 

(2) Criterion-referenced tests; and 

(3) Other examinations of students' skills, knowledge, 
attitudes, and readiness for entering employment 
successfully. 

(c) Results of student ennployment success as 
measured, for exanple, by: 

(1) Rates of enployment and unennployment ; 

(2) Wage rates; 

(3) Duration of enployment; and 

(4) Biployer satisfaction with performance of vocational 
education students as compared with performance of 
persons who have not had vocational education. 

(d) The'results of additional services, as measured by , the 
suggested criteria under paragraphs (aj, (b), and (c) of 
this section, that thfe State provides under the Act to 
these special populations: 

(1) Women; 

(2) Members of minority groups; 

(3) Handicapped person; 

(4) Disadvantaged persons; and 

(5) Persons of Ir ited English-speaking. ability. 



certain what the Office of Education regards as accept- 
able procedures, particularly for assessment of student 
achievement and for evaluation of the effects of addi- 
t i onal ser vi ces , . 

The new requirements left the States in sore need of 
assistance* The States were faced with the task of 
developing or refining procedures and instruments for 
their program reviews^ as well as developing or refining 
. ...fbl L to assess.-.-S.tudent placement . Readyj:. 

made instruments to assess student achievement existed 
for only few program areas . And, this type of test 
development was. too expensive, time consuming, and 
complex for most States to undertake. Hence, a void 
remains, The Office of Education responded in several 
ways to the need to implement new.^ complex, and costly 
procedures "-among them, sponsoring conferences and 
funding several "how to" proj ects^^^-but States still 
felt a need for ass istance.-'-^ 

The second major interpretation came ii) the form of 
a policy memorandum dated April 24,^ 1979 , issued after 
more than a year and one-half pf internal struggle and 
debate within OE over how prescr ipt i ve to be.-'-^ This 
memorandum was written to help States devise evaluative 
schemes that would permit valid aggregation at the 
national level. The memorandum did four things: (1) it 
defined the term "program" (the entity to be eva luat ed ) 5 
(2) it permited sampling of programs; ( 3 ) 1 1 ' encouraged 
States to adopt a cyclical pattern of evaliiation; and (4). 
it promised that OE would not aggregate into national 
summaries either program review data or results of 
student achievement, although it would review and sum- 
marize results to show a national trend. 



The specif ications in the memoraiidum flow more from 
a concern witr bein;^ able to provide a national picture 
than from a cx:ncern for program improvenient . The section 
which gives States permission to evaluate a sample of 
programs within the five-year period of the State plan 
illustrates this point most clearly. The memorandum 
states: "Each of the six-digit code instructional 
programs must be evaluated within the period covered in 
the approved Five-year Plan. But if a State is operating 
100 auto mechanic programs (a six-digit code) throughout 
the State, it may design a representative sample of such 
programs • • . . "^"^ 

Federal administrators believed that, to make the 
evaluatiorio process, workable, sampling was essential In 
large States with many programs. However, in specifying 
a sampl ing uni t , OE has affected the types of program 
revisions that are possible. In so doing,^ it has clearly 
illustrated that the the methods, degree of rigor, and 
information needed may not be the same for all evaluation 
uses. A national picture made, up of placement rates from 
a sample of programs is _per fee t 1 y . acceptabl e . But the 
method may not be perfectly acceptable for making program 
revisions. For one things revisions can be made only in 
the programs in nthe sample. Some programs may never be 
evaluated, even over the course of five years. Choosing 
a program rather than a school as the entity to be eval- 
uated may mean. 4hat school-wide services are 
overlooked. Some states now choose the school or^ 
district as the entity to be evaluated and pose questions 
about the availability and ut i 1 izat ion in the planning 
process of i nf ormat i on related to student and coiimunity 
interests ancj needs and job market demand. These 



questions pertain iiiir a planning process which is schoci- 
wide and not progr-sm specific. Similarly, efforts to 
eliminate sex role stereotyping or to mainstream handi- 
capped students or to provide-counseLing_ are done school - 
,wide, not program by program. Inf ormat i on on these ac- 
t i vi t ies--which certainly may suggest constructive ways 
to revise the programs-may be better collected by 
sampl ing school s . 

The States Implement the Requirements 

Before the 1976 Amendments , evaluat ion of vocational 
education programs in most States was done on an informal 
and unsystematic basis. Few, if any, States had eval- 
uation procedures as complex or comprehensive as those 
described in the ~re g u 1 a 1 i o n s Ne_ve r t h e 1_^ s s , p i e c e s of 
t he eva luat i on sys tem env'isioned by the regulations were 
in place. For example, most States reviewed, local 
programs , al though often no't as part of a formal program 
evaluation. Student aqhievement was measured at the 
local level by teachers and^ for some occupations (e.g., 
practical nursing), by State licensing or certification 
examinations. In many States, student placement darifa 
were col lected, again of ten by individual teacher;^ and 
not: i a manner rigorous enough to yield relia/6le and 
valid results. When the 197*6 leg:islation passe 
began to draw together and systematize many 
procedures. 

NIE exatrined the States" response to the evaluation 
requirements at two points--the spring of. 1978 , -^- ^ and the 
19^9-80 school year.^^ In the spring of 1978 , infor'^ 
matio'n was collected for all States from written 

' :.; . ; -13- : . 
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descriptions of procedures, where they existed, and from 

personal interviews with State evaluators at a conference 

on evaluations sponsored by OE's Bureau of Occupational 

and Adult Education (BOAE). In the 1979-80 school year, 

' * ' °" s °° 1 1 ec t ed__ i n ..con junction : with case- 

studies conducted in 15 States. 

Implementation as of Spring 1978 



By the spring of 1978 (when the regulations liad been 
ouH for approximately six months but the policy memor- 
andum had not yet appeared), work was well underway in 
most States. Evaluation procedures were being extended 
and, in some cases, developed, in accordance with the 
regulations, but States were overwhelmed at the prospect 

° L ... 1'?P.I®E.^.':'..LL".I !^ No t only we r e_ t he., p r o.c e d.,u.r.e.s_-g.LV.e n._. 

in the regulations detailed, complex, and costly to im- 
plement, but during the first year no Federal money was 
appropriated (under Sec. 102(d)) for this purpose. De- 
velopmental work was,, therefore, proceeding entirely at 
State expense, 

, j. 

. Of the four categories specified by the regulat ions-- 
planning and operational processes, student achievement, 
student employment success, and services to special popu- 
lations--most effort by the spring of 1978 had gone , into 
systepiatizing reviews of planning and operational 
processes and assess ing student placement,- an aspect of 
student employment success. Reviews of planning and 
operational processes, or "program reviews" as they: are 
usually called, were being conducted Jn 13 States^ and 
_b;e±ng_r_e-v-i-sed_-i n^ - -an o4;he-r. -3-7-.--— As-s-es-sme n t s— o f— s-t ude nir- 
placements, or student follow-up, were being done in 20 
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states and being revised or developed in another 25. Far 
less attention was b,eing given to the systematic 
statewide assessment of student achievement because this 
procedure requires some development of instruments. 
Almost every State, therefore, decided to continue for 
the time being with assessment by teachers and to wait 
and see what other States would do, 

Implenientation of School Year 1979-80 



By school year 1979-80, the developmental work on 
program review procedures and follow-up had been com- 
pleted in most States. Many States were then. faced with 
the task of compressing their program evaluations into 
the last three years 6f the fivp-year cycle of th^ legis- 
lation. The most consistently implemented category in 
the 1 5 c a s e - s f u d y States wa s a p r o g r M r e v i e w 6 XT p I a n n i n g 
and operational pr ocess es ,' typ ica 1 ly ihvo'lvihg a "self- 
evaluation" by the school and an on-site visit by an 
external team. • In only 2 of the 15 StateTs wer^e program 
reviews still being conducted by individuat-^a 
specialists, rather than as part of a coordinated, 
statewide process. This high level of implementation was 
not surprising s-ince the program review process fit com- 
fortably with the technrcLgi assistance role which State 
education agencies were accustomed to playing with local 
agencies. The usefulness of reviewing planning and oper- 
ational processes was we 1 1 establ i shed , aiid \the /moni- 
toring of local programs was not new. What the 1976> 
Amendments did was to cause^ States to become more 
systemati.c in this undertaking and to expand re v i ews to 
address some new issues. 



FoLlow-up of students to ascertain job placement was 
the next most firmly implemented evaluation activity. 
Here again^ the'utility of the procedure w 4 generally 
established and the procedure itself of-ten ^ place,, 

a^t least informally, pr ior to Federal leg! _ The 

l^Sl^lat ion---both the evaluation provisi .n Sec. 

112(b) and the requirement that these data b luded in 

the Vocational Education Data .System--proviuu :he stim- 
ulus for systematizing the activity at a statewide 
level. By the school year 1979-80 , 12 of the 15 case- 
study States h^id in place a statewide system of re- 
porting; in the remaining 3 States, such a system was 
still under development. While the actual collection of 
data was generally a local responsibility. State agencies 
often facilitated the process by providing forms and, in 
many cases, the names of the students to be followed up. 

Most of the States fol low up both completers aad 
leavers, as required by the Act. Howe^<7er , there are 
ser ic-is pract i cal d i ffi cult i es i n gat h^^iring accurate and 
reliable data which are cons i s tent across districts. 
Th^ese difficulties are compounded 'by administrators' lack 
of conviction that such standardized fol low-up data are 
necessary or appropriate and by the lack of Federal fund-, 
ing to ease rhe burden which the req^'ii rements place on 
State and local resources. 

Formal xol Low-up data on employers' satisfactibn are 
collected mnnh less co'nsistently than data on student 
placement. ^PP^^oximate ly^^^^h^^^^^ of the case-study States 
do not have ax^s tatewfde employer fol low-up procedure at 
both the secondarjr^nd postsecondary levels. Vocational 
educators are less -incl ined to see employer -f-o-How-up as 



oan important and necessary indicator of vocational edu- 
cation effectiveness than, they are student follow-up; nor 
are th^ data required for VEDS. At le^st one of the 
case-study States does hot plan to survey employers 
because it has data which shew that t^ie employers' 
opinions are highly cor related with students' reports of 
empioynient- success . — Given this strong relat ionship, the 
State argues, separate measures of em^loye~r^~* "sa"^^^ 
faction are duplicative. In those States which do 
collect employer follow-up data, poor response rates are 
an even bigger problem than thev are for student follow- 
up data. 

Student ^achievement is generally regarded as an 
important index of program effectiveness. Nevertheless, 
it is generally believed that collecting th i s inf ormat ion 
is best carried out locally by individual teachers, 

-rather than ^^^^^u^^^^^^ on a statewide basis. Before en- 
actment of the 1976 legi s la t ion, none of the 15 States 
required statewide reporting of student achievement data, 
and State respons ib i 1 i ty was~ seen as the mon i tor ing -o f 
local activity. However, in response to the legislation, 
most of the 15 States have tried to systematize the as- 

' sessment' of stoident achievement to .some extent. For 
example, some States, individually and wi,th others in 

.consortia, are working to develop occupational profi- 
ciency standards and criterion-referenced curricula for 
vocational programs. The* process is far from completed, 
for it is^ complex, time-consuming? and rife" with tech- 
nical difficulties. ^ Some States fear that statewide 
achievement testing or proficiency cr iter ia. would bring 
uniformity and regimentation in curricula. 



nondiscriminatory in 
origin. A few States 
sponsible for special 



- .. . Less attention has been given to the fourth require- 
ment laid down by the'^ regulat ions — evaluating the "results 
°- ^-^^lUonal services-to -spec iar heeds populations. The 
program rev iew process typical ly examines the attention 
given to special needsl popu 1 at i ons , but focuses on access 
to, not on the resjus ,of, vocational programs and 
servie^s^ Some States! ask, for example, whether-supple-- 
mental services are p^^^^ to handicapped and disadvan- 

taged students and whether instructional materials are 

terms of sex, race, or ethnic 
have education staff members re- 
needs popu lat i ons /and sex equity 
participate in the program review teams. Other States 
argue that the additional services to special populations 
are automatically included in the regular evaluation 
procedures, and that no special attention is ,/ therefore , 
needed. ' ' 

- y>.. • . - 

Overall, hbwever/, states are engaging in a consid- 
-erable amount . of- ..evaluat i ve activity. There is great' 
var iat ion in its scope and depth, but the States" -are 
demonstrating a strong eonmitment To and a significant 
investment in fulfilling this requirement. 

The Use of Evaluation Results 

^Congress intended that evaluat ion results be, used to 
revise and improve programs. Howe,ver.., the way in whi,ch 
they are used may vary, as . do the types of improvement 
they suggest. For example, the action suggested by a 
finding that equipment is out-of-date 'will .differ from 
that suggested by a finding that students are not getting 
jobs. Likewise, the use of findings at the local leVel 
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might differ from that at the State level, and both may 

o 

.-^^JI^A^. J^^^^ level • In any case, eval- 

uation findings are intended to be used, and it is 
important to find out whether they are, and if so, by 
whom and in what way. 

When States, were asked directly in the spring of 
1978 why mcs\ evaluations were^ being done, 32 cited 
program improvement as a reason, although 9 States said 
"th^ main reason they were doing the evaluation was simply 
to comply with a regulation, Thus% the overwhelming 
majority of State officials saw evaluative findings 
linked, to program improvement even, at an early stage of 
implementation. • However, the in-depth look in 15 States 
during the next school year ( 1979-80 ) showed that the 
uses of evaluat ion results varied among the four eval- 
uat^ion categories, much as implementation did, as the 
following findings demons.t rate . . ^ 

The most thoroughly implemented act i vity was . program 
review; it was also the activity whose results were most 
used. In fact', most corrective action came in response 
to findings frgjTi re v i ewed programs . In cory t ras t^^d.ata on. 
student achievement, on the results of services to 
special populations ^ and, in many cases, on employers* 
sat isfaction generally were not available and therefore 
could ^not be used by the. State agency for program im- 
provement or other purposes. . 

' ■ .. ■ - 

Because results of program reviews were frequently 
used, it is important to examine tfTb.e i r content and to 
understand the way in which the i r resul t:S_we r e used.. The 
content of program reviews paralleled, in most cases, the 



type of information that would interest an accrediting 
team. Topics regularly covered include quality of 
facilities, equipment, and curriculum, whether in- 
structors were certified, and whether guidance and coun- 
~seLlng_^services. were available. Only a few States in- 
quired/into topics, that are major Federal concerns, such 
as access of women and other special populations to 
programs, whether labbr market data or student placement 
data inform local planning, and the. like. Getting these 
sorts of topics into' program reviews would be' one way of 
encouraging greater attention to Federal concerns. 

The action taken by the States to correct weaknesses 
uncovered in the program review was an important step* in 
ensuring that impr^oyement occur . The program; review was 
also an important transaction between the State and local 
school districts and one which afforded the States an 
opportunity to ensure that LEAs be attentive to the 
topics the State believed were important If the State 
embraced Federal priorities, these would be conveyed .to 
locals during the program review. The State could com- 
mun i catey -thriiugh program review. Federal concerns to the' 
locals, and, through^-monjjor Ing, they could offer tech- 
nical ass i stanc.3 in respond ing to-these^ concerns . 

After the review, the State informed each disTrTcT 
or i ns t i tut i on of the deficiencies or weaknesses foilnd by 
the review team. All of the States offered technical 
assistance to local agencies in improving prograpns. 
Beyond that, however. States varied considerably in the 
means by which they oversaw the changes dictated by the 
review and^ in che sanctions which they imposed - 'if 
cor r ec t i ve . act i on was not taken. In some States, the 



implementation of changes ^was left to the .discretion of 
the school d i s t r i c t • Othe r States had more formal jpro- 
cedyre^ involving requirements for corrective actiori^, 
^plans,- and later visits from State staff to monitpr the 
implementat ion . of the changes, ^he ultimate sanction for 
failure to take cor r ec t i ve -ac t i on was the threat of; with- 
holding program apprcS^al or Federal fundiiig, ^ However , 
mos t S'tates were extremely reluctant to u^e this* They 
preferred to continue working with the district- in a sup- 
pdrtive way and. to ^encour age progr iam improvement • 

Data on student placement were used, far less fre- 
q.uently than program r.eview data. A'For the most part,. 
States merely aggregated placement rates across distr icts 
"and , passed (^them to the Federal lev.el as part of VEDS*. 
Generally, prbgramnatic dec i s i ons, were, not made on^. the 
basis of placement rates. Only five of the 15 States 
have made this; data ayailabl^e for- -use in the progr>.r 
rev iew .process and in planning. One of these States, for 
example, ' produced for "each program in , each distr ict a 
printout , of student placeme^it data from the previous, year 
together* with Enrollment and labor market projections to 
be used in local planning. 

The preceding^^isctission , deals with use o'f eval- 
uation results tha/state level. Systematic, data 

across States SL^^^ut^use at the locM level are not 
available,' but there is information from one -recently 
.completed study for the State of Illinois. 1*^ This study 
sought to learn from a large number of LEA administrators 
(609 were surveyed, 30? responded) what evaluation 
activities were initiated and how the^ results were 
used . \ Ful ly 73% of the respondents indicated that they 



Four Conclusions Concerning Implementation 

The first conclusion is simply that the message 
embodied in the regulations and policy memorandum is 
different from that in the legislation; that is to say, 
the message changed during transmission. The discrepancy 
between scale of activity suggested in the Act and in the 
regulations is enormous. The Office of ;Education, 
through regulations, gave the States a message which was 
not contrary to but broader than that pointed to in the 
legislation. The dimensions along which programs were to 
be evaluated have been expanded from two narrow areas in 
the Act--student placement and employer satisfaction-- to 
four broad areas in the r egu lat i ons rev i ew of planning 
and operational processes, student ache i vement , employ- 
ment success, and services to special populations. The 
notion that evaluat ions are to be used was not reinforced 
in either the regulations or the po 1 i cy memorandum. 

The second conclusion is that JJie__ejjr_a.i.uat4on re^^ 
xj-U-i-temen t-s—S't-imui aTed much activity. Imp 1 emen t a t i on has 
been uneven across and within States, but States are 
demonstrating a s t r ong connmi tment to and investment in 
these requirements. States also differ in their ability 
to implement the evaluation requirements. Even more 
pronounced are the d i f f e rences within a given State in 
its ability to respond to the four dimens ions of eval- 
uation stipulated in the regulations. Reviews of plan- 
ning and operational processing are the most fully devel- 
oped evaluative procedures, but most States are moving 
toward implementation of a student follow-up system. 
States are having the most difficulty with assessment of 
student achievement, because appropr iate assessment in- 



struments are not ava ilable for many courses ^oi 
programs. States have made limited progress toward being 
able to assess the effects of programs on special needs 
groups. ^ 

The third conclusion is that even if full implemen- 
tation were to' occur by the desired time, it is question- 
able whether the intention of Congress in introducing 
evaluation requirements would be realized. Virtually 
lost in the procedural activities at both the Federal and 
State levels was the fact that Congress .wanted eval- 
uation results used chiefly to revise id improve 
programs. Yet the emphasis at the~State i eveT~was "^^^ 
getting systems in place to conform to the regulation and 
less on the use to be made of the results. This reaction 
is also not surprising given the fact that the messages 
from the Federal level encourage the emphasis on-pno-'" 
cedure. For ^example , th:e regu lat i ons f rom the Office of 
Education consist only of procedural requirements and do 
not make explicit the array. of benefits that might flow 
from compl iance. The message about utility, about eval- 
uation results being used to revise programs, was not 
reinforced by the Office of Education and was not empha- 
sized by the States, which merely tried to implement the 
procedures as quickly as possible. 

The fourth conclusion is that despite the consider- 
able amount of movement. States miay not fully meet the 
requirements in t'he regulations by the next reauthqr- 
ization period, when performance will be reviewed. The 
prime reason for. this is that change time. During 

two and on^-half years that /it took the Office of 
Education to get the full seV of instructions to the 
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states, the States, had to search for and/or develop 
procedures and instruments, and then, in some cases, 
revise procedures, as new instructions came from the 
Office of Education. Because of the comp 1 ex i ty- of the 
regulations and because most States v/ere not engaging in 
rigorous evaluations when these regulations were issued, 
it will take several years before evaluat i on procedures 
are developed and implemented by most States. Thus, it' 
is safe to predict that some States will not be able to 
meet fully the requirement? in the five years from 
reauthorization to reauthorization. 

The-- story of implementat ion - of — the~eva liiat i on--r-e~ 
quirements is probably typical of the problems associated 
with the imp 1 eme n t ia t i on_^X--S„imi J^a-r— p ravi's^'orf^^ r a 1 

-gran t^ilT- aid legislation. The distance from the Capitol 
to the classroom is great, and there is much opportunity 
for distortion 'of intent in transmission between. One 
can certainly find some instances to the contrary-- 
instances of very faithful interpretation of statutory 
provisions at the Federal and State levels.. One can even 
find instances! of interpretation bringing practice more 

legislative intent, if not language, 
pportunities f or mi sunde r s tand i hg are so 
numerous that some distortion is almost inevitable at 
some level. The timing and the unevenness of the 
implementat ion at the State and local levels is also to 
be expected with requirements like those pertaining to 
evaluation, wh i eh . dea 1 'wi th a technical topiq and which 
call for a considerable amount of change. 



in line with 
However i the o 



The Effects of The Evaluation Requirements 

In enacting evaluation provisions with student 
placement . and employer satisfaction as the criteria 
against which program success was to be measured, 
Congress wias reacting to findings from the GAO report and 
elsewhere; that vocat ional offerings were not in line with 
labor market demand. The 1 eg i s 1 at i on intended to specify 
indicators of labor market demand as criteria for eval- 
uation. Data on these indicators were to be used as a 
basis for deciding whic'h programs to add, drop, or 

, 1 Q 

maintain. The legislation did not imply that placement 
data were t o b e the _ soXe d e-t-e rm ina ni: oT pTogram 

ot-feTTngs. However', the 1 eg i si a t i.on , together with the 

reports from the House . and the Senate, does seem to 
suggest that the ability of students — from pa'rTiV^^ 
programs to f ind jobs w^^^^^ be a factor in deciding which 

' 1 programs to support. Thus, the legi s 1 at i on i s seeking to 
influence program offerings with process requirements. 
With this scenario in mind, it is appropriate to ask 
whether the evaluations have had the intended effect. 

The answer to the first question is "yes," but not 
exactly in the way Congress envisioned. Evaluation 
• results are being used to revise programs, but not 
according to the eKpected scenario. That is the results 
used came not from student placement but from the 
findings of program review-, a process called for in tJie 
regulations but net in the legislation.. Usually this 
process involves {i visit by some external review team 
either nominated by or. composed of State 
admiriistrator,§'. Program strengths and weaknesses are 
noted and communicated to the local institutions, 
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followed by vary ing (dependi ng on the State) amounts of 
assistance ot mon i tor ing - to ensure that revisions are 
made. This is a comfortable role and function for States, 
to assume and locals to accept. The types of program 
revision made, of course, depends upon the type of items 
investigated in the review. In most States, the review 
team invest igates. topics such as quality of equipment and 
faciliti'es or whether teachers are certified, and 
suggests improvement within these areas, , Programnatic 
revis ions ' based . on this information, whi le useful , do 
little to further Congress' purpose in requiring eval- 
uation, which is to bring programs offered more in line 
wi th labor market condi t ions • A few States ask as part 
-of- program reviev/s how the institution decided which 
programs to support. Some States ask further whether 
student placement data were used to arrive at these 
decisions. This is closer to the legislative purpose. 
The review team can ask about the bases upon which 
planning decisions are made and can ensure that labor 
market demancl f igures - heav i ly in the considerations, A 
few States, also, investigate aS' part of their prograim 
reviews other ^topics of concern to Congress, such as 
whether the needs of special populations are being met or 
what is being done. to increase access to women and minor- 
ities. Since locals are accustomed to being reviewed by 
the State and making revisions based on the reviewers' 
coninents, this is a process that could be used to give 
Federal concerns a wider dissemination and emphasis. 

While the legislation is clear- that the major use of 
evaluation is for i.local program improvement, it also 
ca 11 s for a second use--nat i pnal repor t ing*. — St udent^ 
placement data collected locally are entered into the 



VEDS system and aggregated, at the - State and Federal 
levels to form a national picture. It is worth noting 
that the degree of stat i s t ical r igor and the need, for 
compa tabi 1 i ty of data are greater for t,his use than for 
the program plann i ng-program improvement, uses. Data 
collected at the local level and aggregated first across 
districts and then across States must.be compatible from 
place to place. Definitions of terms (such as completer, 
leaver, related to training) must, be constant, and the 
sampling procedure used must be consistent. These re-, 
quirements do not hold with the same rigor for data used 
for program . improvement purposes wi thin a given 
district. Fur thermore , the requirement for one purpose 
may be in conflict with requirements for another. For 
example, a sample of. school programs must be random to 
serve the needs of national reporting; on the other hand, 
a non-random group of the schools or programs most in 
need of impr^ovement might be prefei^red for program im- 
provement purposes. 

Since one evaluative process may not be- able to meet 
all needs, it may be necessary for policymakers at the 
Federal and State levels to review the ^ potent ial use of 
eva luat ion--f or program improvement at the local level, 
for planning and funding decisions at the State level, 
for producing national data--and make <^ choices among 
strategies to emphasize particular uses. 

The tens ion between evaluation for program improve- 
ment at the local level and evaluation for national re- 
porting is by no means unique to vocational, education. 
The author i z i ng . leg i s 1 a t i on for pther_ programs- carries 
with it a requirement to evaluate. For example. Title I 



of ESEA has required program evaluations since its incep- 
tion in The legislative history of ESEA reflects a 
strpng congressional interest in obtaining evaluation 
information that is useful* for program improvement at the 
local level, but evaluative information has also been 
used for reporting purposes at the nationaT level. The 
tension between conflicting demands of data at two levels 
has been noted and the way. in which Ti tie I evaluations 
were used at the local level has been i nves t igated . •'•^ 
The i nves t igat ion found that the primary function the 
evaluations serve has been to meet Federal and state 
reporting requirements; evaluations have not served pri^ 
marily ^as a means of judging the program or as a guide 
for program improvement. 

The lesson is not only that one evaluation process 
may not be able to serve all needs, but also that choices 
may have to be made among the various uses of evaluation 
and that those chosen uses must be properly emphas ized at 
the Federal and State levels to ensure their 
realization. If program improvement is the prime goal, 

this:must be emphasized, even at the expense of gathering 

/' * ,. . . ' ' ' 

nat ional data. 1 / 

The uses just described are not the primary ones 
envisioned by the legislation. Rather, the legislation 
calls for a collection of student placement and employer 
sat i s fact ion data, and intends for these to be one; factor 
in deciding program offerings. Student placement data 
are indeed col lected, but are by and large not used to 
determine program offerings. The reasons for -this are 
provocat ive and -important- 



One reason is that the decision-making process is a 
political one. DetJisions on whether to continue a 
program are influenced by cornnunity pressure, student 
demand, presence of a' tenured tieacher, and, to a lesser 
degree, by data. In fact, at the secondary level, 
student demand is likely to be the primary determinant of 
program offerings.^" In catling for student placement 
data and iji encouraging their use in program ' plann i ng , 
Congress is super impos i ng a rational process on a polit- 
J'^^ effects of the rational' process are apt to 
be at the margins'. Federal requirements can and do 
affect activity locally, but they cannot transform the 
whole enterprise. 

A second reason for limited use of placement rates 
in determining program offerings has to do with the ap- 
propriateness of the criteria. At the State level, 
placement rates are not appropriate as the sole 'a<;ter- 
minant of which districts or schools a State should 
support. Low placements can indicate either that a given 
schopl is preparing students poorly or mere'ly that the 
school is in an economically depressed ar^^a. Obviously, 
one would not want placement rates to lead .to a decision 
adversly affecting the latter school. Placement rates 
can, however, signal problems to State administrators, 
who can then give the situation a closer look. Placement 
rates at the program level within a school or district 
are more telling figures, but still only signal a problem 
and do not reveal its source. All students who stay in 
the ■ area are presumably similarly influenced by its._ 
, cond^^ t ion. - Lower-than-average placement ; rates 
for a given program under these condi t i ons- may indicate 
one of three problems: that the program is not in tune 



with the labor market needs of the area, that the program 
is of poor quality, or that the program ha.s larger-than- 
avera^e enrollment of- disadvantaged, handicapped, and 
other hard-to-place . student s . Given that low placement 
rates may be caused by a variety of conditions, the rem- 
edies likewise vary. For this reason , programmat i c 
decisions should not. flow automatically from a placement 
rate. Furthermore, the fear on the part of vocational 
educators that r igid ins i stance on placements might set 
up pressures to exclude! the hard-to-place is real. 

In short, student placement rates are indeed a use- 
ful piece of information to inform program planning, if 
they are used sensibly. Used in isolation, they may give 
misleading results; used -in conjunction with other infor- 
mation, they richly inform the planning process. 



FOOTNOTES 



U.S. General Accounting Office. Wha_ t is the Role, of 
Federal A ssistance for Vocat i onaj Ed ucationT 
Comptroller General 's Report Congress! 
(Washington, DC:, U.S. Government Printing Office. 
December 1974). /- 

• U.S. Congress, House, Committee on Education and 
Labor, The Vocational Education and National 
Institute o f Education Amendments of 1976 ; Report 
_tg_ Accompany H.R. 12 835^. House Report No. 94-1085, 
94th Cong., 2nd Sess., May 4, 1976, p. 20. 

• U.S. General ; Account ing Off.ice, cited in House 
Report, op. ci t . , pp. 20-21. 

The major portion of , the research for this study was 
conducted before the creation of the Department of 
Education, at a time when, responsibility for 
vocational education rested with the Bureau of 
Occupational and Adult Educa t i on (BOAE) . For this 
reason these ,t i t les , rather than the current ones-- 
Department of Education and Office of Vocational and 
Adult Education (OVAE)--are used throughout this 
paper . ^ ■■ 

U.S. General Accounting Office, cited in House 
Report, op. ci t . , pp. 20-21. 

Drewes and Katz, cited in House Report, op. cit.. 
p. 21. ^ — ' 

In addition, the Comisioner of Education was 
required to conduct yearly analyses of each State's 

^ , plan and , accountability report, including 
"suggestions for improvements in the State's 
programs ( Sec . 1 1 2 (a) ( 1 ) ) ; and the Bureau of 
Occupational and Adult Education was required to 
conduct- a review ol' "strengths and weaknesses" of 
federally assisted programs in at least 10 States a 
year. (These . are known. asv- MERC/Q--Managemerit 

■ Evaluat ion Review for :compl iance/Qual i ty. ) Other 
_-!:, requi rem'ents f or evaluat ion i n the Act include those 
charging the National and State Advisory Councils 
, f or Vocational Education with conducting iindependent 
evaluations of vocational education programs, 
services and activities. This chapter presents only 
-a discussion on the effects of the requirements for 
State evaluations. . ' ' 
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House Report, op. ci t . , p. 38. 



House Report, opl ci t > , p. 38, states: 

The present Act encourages. States to conduct 
per iodic evaluat ions of local programs and to 
use these evaluations in revising their 
programs. The Conmittee's hearings showed that 
this provision seem to be having little effect. 

The Conmittee's bill (Seotion 108), therefore, 
conta i ns new requi rements concerning evaluat ion 
of programs. Specifically, within the' three- 
year period of the. State plan each program 
within a State receiving assistance under the 
Act must be evaluated as to its ef f ect i venefss . 
The evaluations, must, be used to revisf; i:he 
State ' s programs and a summary of them must be 
submitted to the Federal government.' 

J . F. Jenn i ngs and C. W. Radcl i f f e , Conrnentary oh 
Legislation Affecting Vocational Education Research 
and Development" (Columbus , Ohio: The National 
Center for Research in Vocational Education, March 
1977), p. 7. . ^ 

in answering a question on why the particular 
critisria were chosen, one of the authors states: 

The crux of the matter is "how do you tell 
whether a program should be retained or; 
dropped and whether a program is successful 
or not . " \ 

When you attempt' to answer these questions, 
the fundamental criteria must be jotb 
placement in jobs for which an individual ijs 
prepared or in a closely related job are^. 
The next logical criteria relates -to a' 
person's perception of whether or not he/she 
was well prepared- for the job and whether or 
not he/she was satisfied with the area, for 
which he/she was prepared.. Third, one . must 
consider the employer's perception of 
^ " /' vv-hether the vocational program is worthwhile 
from his/her point of view. That perhaps is 



the critical ^ measure. Thus we selected' 
those criteria that should^ be the very 
essence of wTiat you would bas-e a judgment 
as to whether a program should be added, 
dropped or maintained. 

Tederal Re^:i:ster . vol . 42, no. 191. October 1, ' 
1977 ,. \ Sec. 104. 402 ' ■ 



Examples of such pro j ects are^^ the manual s and 
handbooks developed by the National Center for 
Research in Vocational Educat i on under its con t r act 
with BOAE. There have been 13 such handbooks, among 
t h em : - " ' 



Eva luat i on Handbook Vo Tume I* : 



Gu i de 1 i nes ' and 



Pjactices for Fol low-up of Former Vocat ional 
Students ; , ' 



Evaluation Handbook" Vo 1 ume 1 1 : 
Practices 



Pppu la t i ons ; 



for ^follow-up Studies 



Gu i de I Laes 



and 



of 



Sd e G la 1 



ocat ional Education Outcomes; Perspective \f.pr 



Cduca.t i on : 



Guidelines and Practices for Measuring Employer . 
>ati sf act ion with Fo-pmer Vocat ional Students ' 



ra 1 n 1 ng : 



A study by the National Adv^ory Qounc i 1 : for 
Vocat ional' Educat ion also found^^^at OE ' s^ as s i s tance 
''had not been deve 1 oped to i ts^otent i a 1 . National 
Adv.isory Council' for Vocational Education, A Study 
of the Administrat ion, Ope rat ions and Program 



Services 



of 



Vocat ional -Technical 



Educat i on 



Execut i ve , Sunnmary. TWashington, D.C. : U.S. Office 
of 'Educat ion,- Bureau of Occupational ano Adult 
Education, December, 1978), p. .9. 



"Policy Memorandum." BO^E DSVPO - FY 79-2 



Work performed under contract by- CRC Educat ion and 
Human Development , Inc. 

Information was obtai ned .f rom the case studies in 15 
States by Abt Associates', Inc. 



34-- 



T. L, Went ling arid VV. E. Pi land, A Study of Local 
Evaluatfon Practices, in Vdca t ional'^Educat lo nT 
paper ^ presented Tt tlxe Amer ican Voeat ional 
Association. Convention. New Orleans, Louisiana, 
December. 19 80 ) • 

J.F, Jennings and C.V^. Radciiffe, op>^ ci t . p. 7..' 

Jane L, DAvid; Local Uses of. Title I Evaluations > 
Report to the Office of the Assxstant Secretary for 
Planning and Evaluation, HEW, ' (Menlo ParR, 
California: SR 1 Project URu - 6854, July, 1978). 

H. Starr ,C. Maurice, H, Merz,' 'and G, Za'hniser, 
Coordination' in Vocational Edu ca tion' Planning; 
Barriers and Fac i 1 i tator s , fCoTumbus,^ Ohi^o: The; 
National Center for • Research* in ' Vocational 
Education, 1980), p. 22. 
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